fix(db): correct the UTC skew and retire the stale outbox backlog - #245
Conversation
Follow-up to 20260818003000_notification_retry_bounds, which had two problems that only showed up against production data. The backfill used CURRENT_TIMESTAMP. That returns the session's local time, and the database runs Europe/Berlin while Prisma stores and reads these columns as UTC, so every existing row landed two hours in the future and the poller stopped selecting any of them. Confirmed on the server: TimeZone Europe/Berlin, next_attempt_at 00:18:40 local for a migration that ran at 22:18 UTC. The symptom was the absence of a symptom, no delivery attempts at all where three were expected. The column default stays as it is. CURRENT_TIMESTAMP is what Prisma emits for @default(now()), changing it would read as schema drift, and Prisma supplies the value on every insert so the default is never reached. The second problem was what the retry bounds found waiting: 19 undeliverable applicant DMs going back to 2026-07-03, against a batch size of 25. The outbox was six rows short of starving every newer notification, and had been filling up for six weeks without anyone noticing, because a stalled outbox looks exactly like a quiet one. Those rows are retired unsent rather than made due. Delivering them would have sent six-week-old status notifications to applicants who have long since moved on; the status page carries the current state at any time, so a very late nudge is worse than none. Written as an age rule rather than a list of ids, since it is the policy that matters, not tonight's 19 rows.
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
Fixes the UTC skew from the 20260818003000_notification_retry_bounds backfill by recomputing next_attempt_at in UTC (now() AT TIME ZONE 'utc') instead of CURRENT_TIMESTAMP, which had pushed every queued row two hours into the future and stalled the poller. Retires unsent notifications older than 7 days (setting read_at and a last_error note) rather than delivering stale status DMs, then makes remaining unread rows due immediately by resetting next_attempt_at to created_at. Leaves the column DEFAULT untouched to avoid schema drift against Prisma's @default(now()).
No blocking issues surfaced. 4 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 0 functions depend on the 0 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
1 participant
Follow-up to #244. Both problems only became visible against production data, which is the honest summary of why they exist.
1. The backfill was two hours in the future
20260818003000fillednext_attempt_atwithCURRENT_TIMESTAMP. That returns the session's local time, and the database runsEurope/Berlinwhile Prisma stores and readstimestamp without time zoneas UTC. Every existing row therefore landed two hours ahead, and the poller'snextAttemptAt: { lte: now }stopped matching any of them.Confirmed rather than assumed:
The symptom was the absence of one. After the deploy the bot logged no delivery attempts at all, where three failures were expected. Nothing was broken loudly enough to notice.
The rule going in the docs: a backfill of a Prisma
DateTimeusesnow() AT TIME ZONE 'utc', neverCURRENT_TIMESTAMP.The column default is deliberately left as
CURRENT_TIMESTAMP. It is what Prisma emits for@default(now()), so changing it would register as schema drift on the nextmigrate dev, and Prisma supplies the value on every insert, so the default is never actually reached.2. Nineteen applicant DMs had been queued since July
What the retry bounds found waiting:
messagestatus_change19 rows against
BATCH = 25. The outbox was six rows short of starving every newer notification, and had been filling for six weeks unnoticed, because a stalled outbox and a quiet one look identical from outside.They are retired unsent rather than made due. Delivering them would have pushed six-week-old status notifications to applicants who have long since moved on, and the status page carries the current state at any time, so a very late nudge is worse than none. Expressed as an age rule (older than a week) rather than a list of ids, because the policy is the point; it happens to catch exactly tonight's 19.
last_errorrecords why each row disappeared, so this is legible later instead of looking like data loss.Verification
CI runs
prisma migrate deployagainst its own Postgres 16, so the SQL is exercised before merge. After deploy the two counts above should go to zero, andSELECT count(*) FROM notifications WHERE read_at IS NULLshould track only genuinely fresh rows.